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1. ENSURING AND MEASURING VALUE 





1.1. The MCSS — Andersen Consulting Agreement 


Introduction 


The recitals in the agreement between MCSS and Andersen Consulting dated January 27, 1997 
state, in part, that: 


“both Parties are entering into this Agreement on the understanding that all costs incurred by 
them in carrying out the Project and other amounts to which they may be entitled hereunder are 
to be paid from savings generated from the Project, and that while both Parties expect that 
savings will be achieved, there are no guarantees and, neither Party will be paid for any services 
or other costs incurred in connection with the Project until such time as benefits accrue ...” 


This is a performanced-based agreement that anticipates that both parties will diligently 
discharge their respective responsibilities in the pursuit of benefits. In the course of our review, 
there was observable dissonance in the relationship between the parties - Andersen complaining 
that the Ministry has not met expectations for the timely availability of qualified personnel and 
policy decisions to the project; and, the Ministry concerned that the dramatic rise in Andersen 
fees in 1997 has called into question the fairness of the benefit-sharing arrangement and the 
viability of the benefit cap of $180 million in the agreement. 


Notwithstanding the dissonance, the parties are progressing with the critical design phase of the 
project and are exhibiting a willingness to act reasonably and constructively on the issues cited 
here. 


1.2. Expectations of Savings, Benefits Accrual and Returns 


Consonant with Review Recommendation 3 — BTP Business Case Revalidation, the results of 
which are contained in Annex C to the report, we note and acknowledge that the business case 
support for BTP remains strong and that a benefit to the Ontario taxpayer of $346.7 million (net 
of all associated costs), including both economic ($297.2 million) and social ($49.5 million), is 
the expected outcome of the investments of the parties. 


Andersen Consulting assumes a significant portion of the investment financing and related risks 
of the BTP. In assessing the BTP business case, the net benefit to the Ontario taxpayer assumes 
that Andersen has recouped its investments up to the maximum benefit cap of $180 million and 
has fully implemented the deliverables as agreed between the parties within that cap. 


This forecasted outcome, a significant return to the Ontario taxpayer and Andersen Consulting, is 


consistent with the letter and intent of the agreement and appears balanced in terms of the 
relative costs and risks assumed by the parties. 
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In this respect, and in light of the public notice and controversy arising from the recent report of 
the Provincial Auditor, some further comment on the fees charged by Andersen appears 
warranted to, if nothing else, add to the public’s perspective on consultants’ fees in such 
situations in today’s marketplace. 


Much attention has been drawn to the magnitude of Andersen’s professional fees. 


A fair assessment of professional fees should be value-based but, in any case, should be derived 
from the average hourly billing rate calculated from the mix of personnel and rates actually 
deployed on the project. This assures the observer that the relatively high billing rate of an 
Andersen partner, for example, doesn’t mistakenly leave the impression that such rates dominate 
project costs. 


Indeed, Andersen’s current average hourly billing rate of $365.75 based on the mix of personnel 


actually deployed on the BTP at present includes some portion of Andersen’s partners’ time at 
$550. per hour. 


The current average hourly billing rate is expected to fall based on assumptions regarding the 
mix of partner, manager, consultant and analyst personnel that Andersen plans to deploy and the 
anticipated division of labour between MCSS and Andersen personnel. 


Notably, the expectation is that the current average hourly billing rate of $365.75 will drop to 
$261.25 by year-end 1999 — a reduction of 28.5%. 


The margin of profitability in value-based CCP contracts typically ranges from 25% at the low 
end to 35% near the high end. An important influence on this range of margins is the relative 
mix of professional services provided versus hardware/software procurement. Generally, the 
higher end of the range means a relatively high proportion of professional services and vice 
versa. Hardware/software procurement is low risk/low margin business. Professional services 
are relatively high risk/high margin business. 


When the MCSS CPP competition was active, Andersen Consulting appears to have based its 
pricing and concomitant fees on its then current assessment of the project risk and its relative 
competitiveness. As a matter of reasonable conjecture, Andersen’s fees at that time probably 
included margins in the typical range as described above. 


The resultant agreement enabled the escalation of Andersen’s fees to the level of its so-called 
“published” rates. And, since signing the agreement in January 1997, Andersen raised its rates to 
the maximum allowed by the contract. The increment of increase in rates that occurred on 
September 1, 1997 established Andersen’s new rates at the very high end of such rates observed 
in the Canadian market for such services. This action by Andersen may, to a significant extent, 
be attributed to its perception that the risk profile of the BTP changed drastically from the time of 
its original proposal submission in late 1995. 
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The most obvious and, presumably, salient factors in Andersen’s risk assessment would have 
been the significant slippage in the project schedule and the perceived unreadiness of MCSS to 
provide the needed qualified resources and policy decisions to the project in a timely way. 


1.3. Explaining the Delays Incurred by the BTP 


Some of the more obvious reasons cited by both parties for schedule slippage are: 


e The plan to transform the delivery of social welfare programs in the province translated into a 
challenging amount of work to draft new legislation, policies and regulations that took much 
of MCSS management’s time and attention during the first year of the BTP. 


e As the transformation unfolded, more than twenty initiatives, representing separate streams of 
activity, with multiple cross-dependencies and requiring direct management attention, were 
identified. The BTP was only one of these initiatives. 


e It was recognized early on that social assistance reform would result in significant disruption 
to the MCSS labour force. There was appropriate sensitivity to the need to approach the 
human resource management process carefully and in a way that respected the well-being of 
the affected personnel and that optimized their future employment opportunities. This 
resulted in significant delays in the BTP’s access to field operations which, in turn, delayed 
the design and implementation of key early opportunities. 


e .In the course of the downsizing of the Ontario Public Service, early retirement programs 
afforded a number of senior people opportunities to leave public service and opened 
opportunities for promotion to others. The consequent “churn” had a significant impact on 
the BTP. 


e From the outset, the ministry has recognized its obligation to supply adequate numbers of 
resources with the requisite skills and experience to the BTP. Meeting this obligation has 
been a significant challenge for a variety of reasons; the “churn” referred to above and the 
complexity of the recruiting process being two salient examples. The inability of the ministry 
to keep pace with the staffing obligations has, along the way, led to delays in the BTP. 


e The need to assess the continued viability of the business case. On July 8, 1998 Andersen 
Consulting requested a Special Review as permitted by section 2.5 of the Agreement. This 
request was made because of concerns about the costs forecasted and the lower than expected 
level of benefits being generated at the time: 


The project is arguably six months behind relative to original expectations. Delay of this 
magnitude over the life of a four or five-year project is noteworthy but not necessarily 
insurmountable. Andersen, as a worldwide consultancy with major operations in the government 
sector, knows only too well that schedule slippage is a prevalent threat. As such, Andersen has 
no doubt learned how to manage in such environments and how to mitigate its risk. 
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1.4. Mitigating Commercial Risk 


The escalation in Andersen’s fees is a means of mitigating commercial risk by maximizing the 
cumulative margin in the contract as fast as possible. This “Margin Bank” appears to be an 
important part of Andersen’s commercial risk management Strategy. 


This measure is buttressed by the project management approach which, at its core, focuses the 
parties on the cost and benefits attributable to the current task order under consideration or in 
execution. The holistic project perspective is put at some risk in this approach whereas one 
would expect the parties to maintain a shared perspective on the overall project outcomes and the 
overall project return on investment. 


1.5. The $180 Million Cap ( the Cap) 


The Cap in the Agreement represents the maximum aggregate payments to be made to Andersen 
to account for a return on its direct investment, payment of interest charges and incentive 
payments. It was, at the time of contract signing in early 1997, the best estimate of cost on which 
the parties could agree. It was also understood that the parties may mutually adjust project 
functionality, level of effort or their respective responsibilities on a task order or project basis to 
remain within the Cap and to achieve the intended reform. Approved hardware and software 
acquisitions are not included in the expenditure limitations nor are applicable taxes. 


To complete the work if the $180 million Cap is reached beforehand, Andersen could 
conceivably expend a portion of its accumulated margin. A key test will be confirmation that 
the desired scope of work that can realistically be accommodated within the $180 million Cap is 
acceptable to MCSS 


1.6. BTP Outcome (Business Results) 


The current agreement, being performance or value-based, is quite weak in terms of the clear 
specification of the expected outcomes of the BTP. The Provincial Auditor’s overall conclusions 
are particularly pertinent here. That is, the Ministry had not sufficiently defined or established 
the project’s scope and desired business results (outcomes) during the project’s research and 
planning phase. And, since the Ministry had not adequately established the desired business 
results, it could not ensure that such results would be sufficiently measurable to decisively 
determine or agree on progress toward achieving them. The performance of Review 
Recommendation 3 — BTP Business Case Re-validation — has gone a long way to remedying 
these weaknesses. 


1.7. Where from Here? 


To ensure and measure the value of this (Andersen Consulting) service to the Ontario taxpayer, 
the Ministry management has recently taken the important step of revalidating the BTP business 
case. The revalidation process, consistent with Review Recommendations 2 and 3, included an 
exhaustive analysis of the risks confronting the BTP going forward. This risk analysis, unlike 
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that performed prior to the signing of the agreement with Andersen two years ago, had the very 
considerable benefit of past project experience and somewhat greater insight into the design, 
build and roll-out challenges that lie ahead. At this critical juncture and with the design of the 
Service Delivery Model (SDM) design in hand in the August 1999 timeframe: 


Review Recommendation 1 - MCSS — Andersen Agreement — Ensuring Value 


In light of past experience, the parties should, in considering improvements to their agreement 
that better ensure the value of the service, take account of the following recommended 
parameters: 


The results of the revalidated BTP business case including the effectiveness measures 
(business results) expected therein; 


The schedule risks (including those related to municipal readiness and access to third party 
data) exposed by the risk analysis conducted as part of the business case revalidation; 


Within reasonable limits, the government, represented by the Minister, would undertake to 
account for the cost and benefit effects of any proposed legislative or regulatory changes on 
the financial outcomes of the BTP before proceeding; 


Contemporaneously with the availability of statistical proof that improved effectiveness has 
arisen from the Consolidated Verification Process (CVP), verifiable statistical analysis also 
should determine the appropriate benefit duration period and actual dollar savings achieved 
by case type; 


The relatively high probability that the project could require about another three years to fully 
implement; 


The need to reflect the outcome of Review Recommendation 8 —“Value Index” Policy — so 
that the relative values of the contributions of each party to the achievement of BTP business 
results are recognized more appropriately in benefit sharing and so that a balance between 
investment and return to each party is maintained throughout the project; 


Upon mutual agreement to the scope of the SDM following completion of its design, the 
parties firmly commit to its full implementation. Such implementation is expected to be 
achieved within the current benefits cap assuming no further schedule delays; 


A lead role for MCSS managers and project staff in the SDM roll-out phase specifically and 
in change management programming generally; 


Prior to proceeding with the build and roll-out phases of the SDM implementation, a 
readiness assessment of MCSS’ capacity to perform the roles contemplated for it in the 
project plan is conducted and MCSS’ readiness confirmed; 





Hickling Lewis Brod Inc Page 5 


Delays in decision-making or the provision to the project of the necessary qualified resources 
in a timely manner are costly and such costs ought to be borne by the offending party . In this 
regard, consideration should be given to the validity of the benefits cap and of the 2-year limit 
on the benefits earning period following the term of the agreement as effective means of 
recovering benefits lost due to project delays caused by either party. The benefits earning 
period could be lenghthened or shortened or the benefits cap raised or lowered, if necessary, 
based on delays caused by MCSS or Andersen respectively; 


e The estimated six month unproductive delay in the course of the first two years of the BTP 
attributable to the lack of readiness of MCSS to perform its role should be recognized as a 
cost to Andersen in the context of these parameters, 


Performance standards for communications (status reporting, issue resolution, etc.) and 
decision-making (resource deployment, work product approvals, task order approvals, 
acceptance, etc.) should be more aggressive (timing, frequency, escalation, etc.) than 
currently formulated in the agreement; and 


After due notice is taken of any other amendments that may be made based on these and, 
possibly, other parameters, should agreement to the scope of the SDM and to its full 
implementation within the current benefits cap not be possible, the agreement should be 
terminated and competititve proposals sought for the implementation of the SDM design and 
scope acceptable to MCSS. 
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2. ADDITIONAL ACTIONS NEEDED — REVIEW OF PROVINCIAL 
AUDITOR’S FINDINGS 
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2.1. Introduction 


You will find in the following sections, a restatement of each of the Provincial Auditor’s 
recommendations, a report on the consequential actions taken by the Ministry’s management 
associated with each recommendation and, for coherence and ease of comprehension, our 
associated recommendations, if any, for additional actions. 


While we are in substantial agreement with the findings of the Provincial Auditor and the actions 
taken by MCSS management, and acknowledging that weaknesses in MCSS CPP management 
were evident, we are concerned that the issue isn’t so much the relative merits of CPP. Rather, 
the issue is the framework within which MCSS management can fairly report to the legislature 
when the matter at hand — the Business Transformation Project (BTP) -- has everything to do 
with the future effectiveness of the Ministry’s core programs and their delivery. CPP is 
incidental to that framework. 


Since the mid-1980’s, the Canadian Comprehensive Auditing Foundation (CCAF) has provided 
both public sector managers and auditors with a suggested framework for reporting effectiveness. 
We commend the CCAF’s framework to the Ministry’s management as a good means of 
reporting effectiveness. 


It is that sort of framework that helps the legislature to better understand public/private 
partnerships involving risk and reward sharing and the measurement of investment returns over 
long periods in a changing environment. 


At the time of the recent audit, MCSS had not yet crystallized the BTP Blueprint and the design 
of the BTP solution based on which the business case support for the investment could be re- 
validated before going on. Pragmatically, moreover, as happens over the lengthy course of such 
projects in today’s public sector, several senior managers had been promoted or had accepted 
alternative career opportunities. 


Nonetheless, MCSS management and the BTP project team have made significant gains to the 
point of our review and continue to strengthen the foundations for a successful project. Among 
these gains are a good project management framework, a BTP Blueprint and a comprehensive 
Service Delivery Model (SDM) design task order. We also note that the governance of BTP has 
been strengthened by the new Ministry management team in the course of our review. Included 
in these governance changes are, notably: the assumption of the Project Sponsor role by the 
Deputy Minister; the appointment of a new Project Leader; the formal engagement of the 
Ministry Management Committee; the appointment of a Project Director and the recruitment of a 
Contract and Risk Manager. Further, in order to maximize the focus on the BTP outcomes, the 
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Project Director’s mandate will be focussed solely on BTP with the planned change in reporting 
of the Automating Social Assistance Project (ASAP). 


More than anything else of significance in our review, we note with satisfaction that, with the 
new Ministry management team in place and the SDM task order being implemented, the MCSS 
business case has been revalidated to ensure that the return to the Ontario taxpayer remains 
strong and supportable. It is a seminal activity because it should ensure renewed consensus and 
commitment among stakeholders and be the basis for coherent communications and decision- 
making going forward. 


2.2. Provincial Auditor’s Recommendation 1 


For future Common Purpose Procurement projects, the Ministry should adequately document its 
research and planning to clearly demonstrate whether or not the project meets the Management 
Board of Cabinet’s requirements for such projects. 


In addition, the Ministry should consider both other contracting arrangements and maximizing 
the use of its own internal resources. In cases where the Ministry selects neither of those 
arrangements, it should document the reasons for its decision. 


2.2.1. Ministry Response 


The Ministry recognized that Common Purpose Procurement (CPP) should only be used in 
special circumstances, and these led Management Board to approve the Business Transformation 
Project as a pilot. 


The revised CPP guidelines that are being developed by Management Board recommend that 
ministries undertake a readiness assessment which includes a business case, a review of the 
options available to the ministry and a risk assessment. Upon completion of the preparatory 
work, approval from Management Board of Cabinet is required to proceed with the use of CPP. 


Subsequent to signing this CPP agreement, the Ministry implemented stringent document 


management processes which will ensure that decision making is documented and that the 
documentation is maintained. 


2.2.2. Action Taken 


Ministry staff worked with staff in Management Board Secretariat to assist in the development of 
the revised CPP guidelines. Particular focus is being given to areas such as organizational 
readiness and transition and service management. 


2.2.3. Additional Action Needed 


A review of the Revised Common Purpose Procurement (CPP) Guidelines issued by 
Management Board of Cabinet on June 5, 1998 revealed important improvements to the earlier 
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CPP guidelines issued on November 15, 1995. These improvements include somewhat stronger 
guidance with respect to: 


e Organizational Readiness 
e Transition Management 
e Service Management 


each of which benefited from the advice of MCSS officials derived from “lessons learned” from 
BTP. Further CPP guidelines were issued by the Management Board Secretariat on October 30, 
1998 providing significant new detail on whether: 


¢ aproject is suitable for the CPP approach; and 
e the business case can be sustained over the life of the project 


All in all, there appears to remain the need for policy guidance with regard to business case 
development and the inculcation of risk analysis into the development of the business case. The 
relevant BTP risk management regime arises from business case and risk analysis. And, so, 
MCSS has internally adopted the following policy: 





Review Recommendation 2 — Business Case Development and Risk Analysis* * 







Business case analysis ought to be performed by MCSS to identify and select those investments 
most beneficial to the achievement of its defined desired business results (outcomes). Ministry 
officials will adopt the relevant means by which risks can be identified and managed for 
investment projects like BTP both at the point of initial investment decision and at every 
incidence of significant change during project execution. Such a means is the performance of 
risk analysis as part of business case development. Risk analysis is central to business case 
development and any risk-sharing business arrangements. Risk analysis is a particularly valid 
means by which to account quantitatively and transparently for uncertainty and risk in complex 
projects like BTP. 










With risk analysis, it is reasonable to expect that, retrospectively, the risks and financial effects of 
the provision that enables Andersen Consulting to increase its billing rates from time to time to 
the level of its standard “published” rates would have been transparent to MCSS officials and, 
possibly, mitigated before the agreement and executed. 







Because risk analysis ensures transparency, it enables tracking by management and auditability. 
Risk analysis facilitates cause-and-effect-driven quantitative forecasts of all Ministry and client 
benefit and cost metrics — both tangible and intangible. In each case, account is taken of 
uncertainty and risk. 


Risk analysis begins with the specification and codification of the structure and logic model of 


the base case (where we are) and the base case redesign (where we want to be) in relation to 
outcome (business results) metrics. 
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Once done, all cause and effect variables in the model are quantified. Then, both the model and 
the quantitative analysis are exposed to the stakeholders in order to: 


e find stakeholder consensus on the structure and logic model 


e find stakeholder consensus on the median values in the model and the appropriate 
probability ranges for each variable 


e simulate business case outcomes based on the validated model and probability ranges 
e find stakeholder consensus on the business case outcomes 


The risk analysis process flushes out the real and perceived risks allowing management to see 
their mitigation. 


Risk management, the determination and resolution of all threats to the successful achievement 
of project objectives, is the primary concern of MCSS management. From a project’s initial 
planning and organization through design, development and implementation stages, project risks 
should be assessed with due regard to the following categories of risk: 


e Management Risk 
e Political Risk 

e Financial Risk 

e Schedule Risk 

e Contract Risk 

e Technical Risk 

e Forecast Risk 

e Methodology Risk 


or any others that may be germane to a particular project objective or outcome. 


Review Recommendation 3 — BTP Business Case Re-validation sora 


The Ministry ought to apply its Business Case Development and Risk Analysis policy to the re- 
validation of the BTP business case and, in so doing, achieve the following objectives: 


Provide the quantitative framework within which architecture, technology, business process 
and implementation options and choices can be made with due regard for value-for-money 
and effectiveness. 


Provide the framework within which quantitative aims for BTP effectiveness can be 
established as management decisions. 
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Provide the quantitative framework within which quantitative aims for return on investment 
(ROJ) and risk can be established as management decisions. 


Provide the framework within which progress toward effectiveness aims can be monitored. 


Provide the quantitative framework within which returns on investment can be measured and 
monitored. 


Provide the operational business framework within which the MCSS Senior Financial Officer 
(SFO) can monitor financial outcomes. 


Provide the operational effectiveness framework upon which the auditor can rely for 
transparency and effectiveness measurement. 


Provide the framework within which the social benefits of BTP can be measured and 
accounted for in business and policy decisions. 





To quantify and monitor both MCSS and Andersen Consulting ROI. 
Document Management 


The BTP Document Management Process is thoroughly described in a memorandum dated 
October 3, 1997 to project personnel — specifically, Task Order Managers and Functional 
‘Coordinators. 


Review Recommendation 4 —- Document Management * 


A review of the effectiveness of the BTP Document Management Process should be conducted 


by the Director, Information Systems Branch early in 1999 with a report of findings and 
recommendations to the Project Leader - BTP, Project Director - BTPand the Assistant Deputy 
Minister, Business Planning and Corporate Services. 





2.3. Provincial Auditor’s Recommendation 2 


To help ensure that the most cost-effective proposal is selected for all future Common Purpose 
Procurement projects, the Ministry should enhance the selection process by: 


e Demonstrating the risk/reward implications and overall value to the Ministry of each 
proposal; and 


e retaining documentation on completed reference checks Ministry Response 
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2.3.1. Ministry Response 


Consistent with the initial guidelines, the Ministry established the overall value, to the Ministry, 
of each proposal by including overall value as one of the evaluation criteria. At that time, overall 
value was based on a limitation to the amount of costs and risk to be assumed by the Ministry. 


The revised Common Purpose Procurement (CPP) guidelines being developed by Management 
Board recommend a revalidation of a ministry’s business case with the short-listed vendors 
during the selection process. This will further assist ministries and vendors with validating the 
financial and business model with respect to the sharing of risks and rewards at an earlier stage. 


Subsequent to signing this CPP agreement, the Ministry implemented stringent document 
management processes which will now ensure that decision making is documented and that the 
documentation is maintained. 


2.3.2. Action Taken 


Ministry staff have worked with staff at Management Board Secretariat on revising the CPP 
guidelines. Revisions will help ensure that this ministry, or any other ministry considering CPP, 
will be better prepared to enter into a contract. As well, a due diligence phase will be added to 
the process to facilitate the vendor selection process. 


New strict document management standards have been put into place. 


2.3.3. Additional Action Needed 


The recommendation under “additional action needed” to the Provincial Auditor’s 
Recommendation 4 applies here insofar as the retention of documentation on completed 
reference checks is concerned. 


To demonstrate the risk/reward implications and overall value to the Ministry for each proposal 
with a retrospective on BTP but with due regard to future CPP’s, the new MCSS Business Case 
Development and Risk Management policy adopted under Review Recommendation | applies. 


Review Recommendation 5 — MCSS’ Future Common Purpose Procurements 


In its application to future Common Purpose Procurements, MCSS’ Business Case Development 
and Risk Analysis Policy (see Review Recommendation 2) will mean that: 


e The Ministry’s business case would be disclosed in a discussion paper and form part of the 
vendor consultation suggested in Management Board’s revised CPP guidelines; 


The Ministry’s business case (modified, where applicable, to reflect the vendor consultation) 
would be included in the subsequent request for proposals; and 
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e Vendor recommendations that would improve on the Ministry’s business case would be 


evaluated as part of the determination of the risk/reward implications and overall value to the 
Ministry. 





In this way, the differential value of the business case effects of each vendor’s proposal is added 
to the other evaluation factors to assist in the demonstration of the risk/reward implications and 
overall value to the Ministry of each proposal. 


2.4. Provincial Auditor's Recommendation 3 


To help ensure that maximum payments under future Common Purpose Procurement agreements 
are reasonable and commensurate with the value of the work to be performed, the Ministry 
should: 


e assess and compare the reasonableness of the maximum contract payment agreed to 
with the estimated cost of the work that may be reasonably expected to be performed; 
and 


e clearly identify and segregate expected benefits directly attributable to the work to be 
performed under the agreements. 


Also, if the Ministry anticipates that changes to a project’s scope or the amount of work required 
for its completion may require changes to a contract, it should clearly state the criteria to be used 
for determining when such changes have occurred and for deciding how the maximum contract 
fee will subsequently be altered. 


2.4.1. Ministry Response 


Under Common Purpose Procurement (CPP), the parties jointly define the solution to an 
identified problem. Accordingly, the cap on payments negotiated by the Ministry after the 
successful vendor was selected is based on an understanding of the scope of the work 
contemplated in the original request for proposals. The cap on payments is not a guaranteed 
payment to the vendor; it is a maximum investment commitment. An adjustment to scope does 
not necessarily result in an increase to the cap on payments. 


One of the premises of the CPP agreement is that savings can only be attributed to the Business 
Transformation Project if the Project contributed to the achievement of those savings. This 
contribution can include the development of tools and processes required to support the 
implementation of ministry policies. It is through the development and implementation of 
metrics (performance measures) that savings directly attributable to project work are measured. 


In future CPP agreements, the Ministry will include additional criteria and processes for 
determining potential changes to project scope. 
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2.4.2. Action Taken 


As part of the CPP process, a ministry must develop a business case which establishes the full 
range of financial and operational elements required to establish a performance-based business 
arrangement. As well, a due diligence process will be included in the revised guidelines. It 
should be noted that the maximum payment under CPP is not guaranteed payment but is 
dependent upon expenses being incurred. 


In future CPP projects, the ministry will ensure that excluded costs are minimized. 


2.4.3. Additional Action Needed 


As the second anniversary of the BTP approaches and the decision to add a fifth year to the 
current four year contract is pending, it is noted that the Ministry has revalidated the BTP 
business case for its own account including, in light of present opportunities and constraints, a 
full risk analysis. This revalidated business case should become an operational tool for decision- 
support purposes going forward. 


Review Recommendation 6 — BTP Business Case Custodian * 


The Ministry’s Senior Financial Officer (SFO) should be the custodian of the BTP business case. 
The SFO should provide the necessary business case and risk analysis services when Ministry 
management decision support is sought. 





When, for example, any proposals that may affect the overall return on investment to the 
Ministry or to Andersen Consulting are brought forward, the logic and metrics (costs and 
benefits) of the proposed changes would be analyzed as an alternative to the then current BTP 
business case. If the proposals result in an improved return on the Ministry’s investment, they 
would be favourably considered. If not, they’d probably be rejected. 


2.5. Provincial Auditor’s Recommendation 4 


In order to more effectively estimate and control overall ministry payments for future Common 
Purpose Procurement projects, the Ministry should minimize the number of items excluded from 
maximum payment amounts. When items are excluded from maximum payment amounts, the 
items and the circumstances under which costs for them ought to be incurred should be clearly 
defined and estimated. 


2.5.1. Ministry Response 
The Ministry agrees that it is important to minimize the number of items to be excluded from a 
cap on payments. The Common Purpose Procurement (CPP) agreement identifies the items that 


are excluded from the cap on payments. These items include hardware, third party software, 
production support, help desk services and application maintenance. The costs associated with 
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these items will be confirmed at the lowest possible end user cost using the government 
procurement process. 


In future CPP arrangements, the Ministry will more clearly define excluded items and will 
consider including an estimate of costs for those items. 


2.5.2. Action Taken 


None. 







Review Recommendation 7 — Maximizing Return on Investment 1 








In its performance of the business case analysis in response to Review Recommendation 3, the 
Ministry ought to include all project costs. And, in assessing its risks associated with computer 
hardware, third party software, production support, help desk services, annual application 
maintenance costs and any applicable taxes, the Ministry should consider its options to minimize 
cost exposure and maximize the return on its investments. 


2.6. Provincial Auditor’s Recommendation 5 


Future Common Purpose Procurement agreements should ensure that project savings are 
distributed equitably, based on the relative contributions of ministry and consulting staff, which 
will not necessarily correlate with the relative salary levels of ministry staff or the billing rates of 
consulting staff. 


2.6.1. Ministry Response 


The revised Common Purpose Procurement (CPP) guidelines being developed by Management 
Board recommend that an equitable distribution of benefits be established. In future CPP 
arrangements, the Ministry will look at alternatives to the distribution of benefits, in accordance 
with the guidelines being developed. 


2.6.2. Action Taken 
Ministry staff have worked with staff in MBS in the development of revised guidelines for CPP. 


In future CPP projects, this ministry will carefully evaluate distribution of benefits to support 
equitable distribution. 


2.6.3. Additional Action Needed 


It is an accepted norm that some public sector incomes, especially at senior levels, are inferior to 
those of private sector counterparts on a total compensation comparable basis. These differences 
can appear to be especially pronounced when consulting fees are used as the basis for 
comparison. Consulting fees are based on VALUE and are typically constructed from the 
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addition of base salary, benefits, bonuses, overhead and normal margin with the further 
possibility of a risk premium. 


To fairly associate public sector cost-based incomes with consultants’ value-based fees, an 
economic concept of “ Value Indexing” needs to be introduced. “Value Indexing” determines the 
relative value of the public servant which, unlike the individual’s total compensation, is derived 
from factors such as knowledge, experience, leadership, accountabilities, results-produced and so 
on related to the specific business results of the project. 


Review Recommendation 8 — “Value Indexing” Policy * 









The Ministry, in comparing the value of its contribution to a BTP-like project with that of 
consultants when both parties are contributing to project costs in anticipation of proportionate 
benefits-sharing, should develop and apply a “value index” for its personnel reflecting their 
relative value to the project. Should the current agreement be amended, it should reflect such a 
“value index” with its concomitant effect on benefits-sharing. 


2.7. Provincial Auditor’s Recommendation 6 


The Ministry should ensure that all necessary receipts of out-of-pocket expenses to be charged to 
the Business Transformation Project are received and reviewed by the project management office 
so that they can be checked for reasonableness and compliance with relevant expense policies. 


2.7.1. Ministry Response 


The Ministry is committed to accountability and is pleased that the Provincial Auditor brought 
this issue to our attention. Upon being notified of the issue, the Ministry obtained and reviewed 
the appropriate receipts, and they are now kept on site. All receipts will be kept on site in the 
future. 


2.7.2. Action Taken 
Receipts for out-of-pocket expenses are reviewed on monthly basis and are kept on site. 
2.7.3. Additional Action Needed 


None. 
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2.8. Provincial Auditor’s Recommendation 7 

To ensure that the benefits of the Ministry’s Business Transformation Project are fairly 
distributed, the Ministry should include all of its costs related to that project in the project cost 
pool. 


2.8.1. Ministry Response 


The Ministry has now added the costs associated with Change Reporting systems changes to the 
cost pool. In general, costs for implementing new processes will be added to the cost pool. 


2.8.2. Action Taken 


The ministry will ensure that as task orders are developed, project related costs are added to the 
cost pool. Previous costs which were not in the pool but were project related, have been added. 


2.8.3. Additional Action Needed 






Review Recommendation 9 -- Financial Functional Control * 







The Ministry’s Senior Financial Officer (SFO) should exercise functional control with respect to 
the determination, qualification and allocation of BTP costs and benefits. In so doing, the SFO 
should be satisfied that all Ministry units and personnel expending effort or money in support of 
BTP see to it that all qualified expenditures are recorded and charged to BTP. 


2.9. Provincial Auditor’s Recommendation 8 


If interest is charged on non-reimbursed costs for future Common Purpose Procurement projects, 
the Ministry should ensure that amounts paid are based on actual interest expenses incurred. 


2.9.1. Ministry Response 
All Project costs are eligible and reimbursable. Interest has been applied to the complete amount 
outstanding as per the standard billing approach. In future Common Purpose Procurement 
agreements, the Ministry will review alternative approaches to charging interest. 

2.9.2. Action Taken 

None. 
The ministry will consider alternatives in the area of interest charges in future CPP arrangements. 


2.9.3. Additional Action Needed 


None 
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2.10. Provincial Auditor’s Recommendation 9 


The Ministry should ensure that if additional work is to be incorporated into the current or future 
Common Purpose Procurement projects, the benefits of that work as compared with the benefits 
of other alternatives are clearly established. 


2.10.1. Ministry Response 


The Ministry agrees that it is important to evaluate benefits compared with other alternatives. 
When additional work is to be included in a Common Purpose Procurement (CPP), the Ministry 
will assess the various ways of proceeding and select the alternative which represents the most 
value for the Ministry and is consistent with existing contractual arrangements and the need for 
integration into a comprehensive solution that supports the reform of social assistance. 


CPP is a risk-based arrangement in which the vendor invests its own human and financial 
resources with no guarantee of payment. All of the work does not generate benefits. Many 
components of the work will improve administration without generating benefits. Some tasks do 
generate financial benefits and assist with balancing costs and benefits. To date, costs charged to 
the cost pool have been significantly higher than benefits. 


ZAQ 2. Action Taken 


In determining whether additional work is to be incorporated into this or future projects, the 
ministry will: 


e Consider whether the work is within the scope of the contract 


e Consider whether the work can be completed outside of the project without unduly 
adding to the risk of either partner 


e Ensure that the devised outcomes of the project are taken into consideration in 
considering alternate arrangements 


2.10.3. Additional Action Needed 







Review Recommendation 10 — Additional Work so 












In every case where additional work or design options are considered for the BTP, consonant 
with Review Recommendation 2, the benefits and costs of that work or those options ought to be 
subjected to complete business case and risk analysis in the context of the then current business 
case (the base case). In so doing, the available feasible alternatives to the base case should be 
analyzed to determine which yields the most attractive Net Present Value (NPV) or net benefit to 
the Ministry or to the government as a whole.. 


a 
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2.11. Provincial Auditor’s Recommendation 10 


To ensure that task order metrics fairly account for the relative contribution of savings by each 
partner in the Business Transformation Project, the Ministry should ensure that: 


¢ amounts included in the benefit pool are accurately determined; and 


e financial savings attributable to ministry staff adhering to previously established policies and 
procedures are not included in the project’s benefit pool 


Zi lel. Ministry Response 


The Ministry is committed to ensuring that the amounts included in the benefit pool are 
accurately determined. As part of the ongoing metrics development process, a periodic review 
process had been established during the audit which reviews the assumptions and information 
related to performance measures. The Ministry is continuously reviewing the assumptions and 
information related to performance measures and, subsequent to the audit, has already adjusted 
both the baseline used to establish certain metrics and the resultant savings included in the 
benefit pool. 


2h: Action Taken 


Metrics development is an ongoing process. Metrics developed for a task order are subject to 
review in order to determine if previous assumptions continue to be valid. Any adjustments to 
amounts included in the benefit pool are made retroactively. 


In order to ensure that the metrics developed included appropriate assumptions, the ministry had 
a third-party review the process. 


elke. Additional Action Needed 


None. 


In revalidating the BDP business case, tests were applied by the HLB Business Case team to 
ensure that both baseline metrics and the value of incremental benefits attributable to BDP were 
verified. In no case, were benefits attributable to previously established policies and procedures 
included. 


2.12. Provincial Auditor’s Recommendation 11 


The Ministry should take the steps necessary to ensure that the work under the agreement with 
Andersen Consulting supports the delivery of the Ontario Works program and the Ontario 
Disability Support Program with the revised business processes and technology solutions at the 
earliest opportunity. 
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2.12.1. Ministry Response 
The Ministry is committed to ensuring that the necessary business processes and technology are 
in place as quickly as possible to support the delivery of Ontario Works and the Ontario 
Disability Support Program. 


The Business Transformation Project has already successfully supported the implementation of 
several new business processes. 


25122: Action Taken 


The thrust behind this project is to ensure that delivery agents have the tools required to support 
the reform of the social assistance system at the earliest opportunity 


212.3; Additional Action Needed 


Review Recommendation 11— Effective Program Delivery * 





The Ministry management team ought to significantly strengthen the governance and control of 
BTP full expecting that the specific measures set out in Review Recommendation 14 will hasten 
the efficient delivery of needed support to the Ontario Works and Ontario Disability Support 
Programs. 





2.13. Provincial Auditor’s Recommendation 12 


In future, the Ministry should fully establish the type of consulting expertise it requires prior to 
the competitive selection stage to help ensure that selected proposals offer all the needed 
services. 


To be able to demonstrate the basis on which a successful vendor was selected and that the 
selection and that the selection process was fair, the Ministry should prepare and retain rating 
sheets or other documentation for every proposal received and evaluated. 


In instances where it is deemed necessary to sign or revise an agreement for more than the 
proposed fee, the Ministry should clearly establish and document the reasonableness of the 
revision and the basis for its approval. 


O74 (EA IE Ministry Response 


The Ministry agrees that the need for consulting services, initially and for any revisions, should 
be clearly defined and that the documentation regarding the resulting process is retained. 


Subsequent to signing this consulting agreement, the Ministry implemented stringent document 


management processes which now ensure that decision making is documented and that the 
documentation is maintained. 
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2.13.2. Action Taken 


In future, the Ministry will determine the level and type of consulting expertise required to 
support the selection process in CPP. 


As well, the Ministry will adhere to the stringent document management processes now in place. 


Hd if ep Additional Action Needed 


None. 


2.14. Provincial Auditor’s Recommendation 13 


To protect the independence and objectivity of the Quality Council’s work, all members of the 
Quality Council should be free of any actual or perceived conflicts of interest. 


2.14.1. Ministry Response 


The Ministry will ensure that there is no real or perceived conflict of interest on the Quality 
Council. 


2.14.2. Action Taken 
A Ministry representative will be added to the Council. 
2.14.3. Additional Action Needed 


None 
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3. MANAGING THE CPP AGREEMENT 


Se 


3.1. Introduction 


The adoption by Ontario in 1995 of a Common Purpose Procurement (CPP) policy direction as a 
new method of procurement for goods and services related to information technology (IT) — 
based solutions, business process reengineering, outsourcing, etc. is consistent with the choices 
being made in other jurisdictions in response to the public sector change dynamics of the 1990’s. 
These dynamics have become the mantra of public administration generally and the management 
of programs and delivery of public services specifically. They include, for example: deficit 
elimination, debt reduction, public service employment cuts; decentralization of program 
delivery; improved service to the public; and, better targeting of program benefits. Some market 
effects of these changes are noteworthy. They include: 


e Some loss of competencies and experienced public servants arising from downsizing 
and early retirement programs; 


e Reduced capacity to recruit and retain key personnel in high employment/high wage 
functions such as information technology ; and 


e Increased reliance on public/private partnerships to maintain or improve on service 
levels and program effectiveness 


It is these very dynamics that have given rise to CPP and variations thereon, for example, in the 
Canadian ( see Annex A) and the United States (see Annex B) governments. 


3.2. Comptrollership (Measures to Strengthen) 


Late in 1997, the Canadian Comprehensive Audit Foundation (CCAF) published the findings 
and recommendations of its Independent Review Panel on the Modernization of Comptrollership 
in the public sector. We observe that the issues and the environment confronting the 
management and staff of MCSS appear to be quite consistent with some of the Panel’s key 
observations. 


The Panel, made up of eminent Canadians from the private sector, noted that: 


“Executives and general managers, and their support specialists and professionals throughout 
government, are being challenged in all areas (accountability, service quality and efficiency, 
responding to change, stewardship and public trust and principles of fairness and equity in 
dealings with all Canadians). In particular, they cite the following as areas that increasingly 
demand managerial excellence: 
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e the accomplishment of solid results using new delivery approaches and working with 
fewer people; 


e identifying priorities and assigning scarce resources in support of them; 


e adopting and deploying new approaches to maintain meaningful control and 
accountability both in their own ministries and when using alternative service 
delivery arrangements; 


e ensuring that appropriate values attend these new arrangements; 
e managing, receiving and using meaningful and reliable performance information; 


e reporting, explaining and being accountable for results as well as processes and 
inputs; 


e matching more creative and client-driven decision making and business approaches 
with solid risk management; 


e creating an environment in which taking risks and the consequences of doing so are 
handled within a mature framework of delegation, rewards and sanctions”. 


Ultimately, these matters are also of concern and interest to ministers and the legislature. 


The federal Auditor-General, in commenting on innovation and sensible risk-taking in the public 
sector said, in his most recent 1998 report to Parliament, in part: 


“There are forces, however, that many observers believe militate against risk taking and 
innovation. These include the effects on public servants’ morale of such interrelated factors as 
downsizing, the long wage freeze that ended only last year, and a perceived lack of support. At 
the same time, more openness and a greater level of public accountability for results can 
sometimes unleash undeserved criticism when results fall short of expectations. In these 
circumstances, many public servants may feel the most prudent course is to play it safe.... We 
— the public, parliamentarians, public servants and this Office — have to work together to help 


remove the constraints and find ESE to ee the innovation and responsible risk taking that 
are required now more than ever.’ 


3.3. BTP Effectiveness 


The management of the Ministry of Community and Social Services (MCSS) will be called upon 
to make representations on the effectiveness of BTP throughout its development and operational 
life. These representations will, at different times, be made to the Minister, to Management 
Board of Cabinet, to the Ontario Legislature, to the media, etc. BTP will also be subject to 
periodic audit by the Provincial Auditor who, in the absence of any other standards against which 
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to audit for compliance, will rely on MCSS management’s representations. In so doing, the 
auditor will form opinions as to the fairness of the information that management has reported. In 
particular, the auditor will report to the legislature on instances where there has been a lack of 
due regard to economy and/or efficiency and where there are inadequate processes for measuring 
and reporting effectiveness. 


Review Recommendation 12 — BTP Effectiveness Reporting ** 


The management of the Ministry ought to be ready to make representations on the effectiveness 
of BTP throughout its development and operational life. It is recommended that MCSS adopt the 
following Canadian Comprehensive Auditing Foundation -- suggested effectiveness attributes for 
BTP: 


Management Direction: the extent to which the objectives (outcomes) of BTP are clear, well- 
integrated and understood, and appropriately reflected in MCSS’ plans, structures, delegations of 
authority and decision-making processes; 


Relevance: the extent to which BTP continues to make sense in regard to the problems and 
conditions to which it is intended to respond; 


Appropriateness: the extent to which the design of BTP, and the level of effort being made, are 
logical in light of the specific objectives to be achieved; 


Achievement of Intended Results: the extent to which BTP objectives have been or are 
expected to be realized; 


Acceptance: the extent to which the stakeholders in BTP judge it to be satisfactory (to their 
needs); 


Secondary Impacts: the extent to which other significant consequences, either intended or 
unintended, positive or negative, have arisen as a result of BTP; 


Costs and Productivity: the relationship among costs, inputs and outputs resulting from BTP; 


Responsiveness: BTP’s ability to adapt to changes in such factors as case volumes, program 
design, available funding, technology, etc.; 


Financial Results: the matching of, and accounting for costs and benefits arising from BTP and 
the accounting for and valuation of assets and the return to the Crown on BTP investments; 


Working Environment: the extent to which the BTP provides a productive work atmosphere for 


MCSS employees, appropriate opportunities for their development and success, and promotes 
commitment and initiative; 
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Protection of Assets: the extent to which important assets — such as intellectual property, 
sources of supply, key personnel, important records and information, software, etc. are 
safeguarded so that BTP is protected from the danger of losses that could threaten its success, 
credibility, continuity, etc.; 










Monitoring and Reporting: the extent to which key measures pertaining to performance and 
organizational strength are identified, reported and monitored; 





To portray BTP’s effectiveness in a balanced way, MCSS management should adopt all the 
above attributes so that third parties, especially legislators, can form a balanced judgement on 
effectiveness. 


3.4. BTP Project Governance and Control 


BTP is not simply another information technology project. Rather, BTP is the new business 
process for Social Assistance Program delivery in Ontario. It is an enabling element to greater 
program effectiveness including implementation of single tier program delivery. BTP will 
enable stronger service level performance and contribute to MCSS efficiency goals. Information 
technology is incidental to these outcomes. BTP is integral to the core business of MCSS. 


Review Recommendation 13 — BTP Governance and Control * 


The work of BTP ought to be integrated within existing MCSS processes and structures. Such 
integration means that, for example: 


The ADM, Business Planning and Corporate Services will exercise full functional control 
and provide support services in finance, human resource management and contract 
management; and 


The ADM Program Management will exercise full functional control and provide support in 


operations during the design, build and, especially, the roll-out phases of the BTP; 


The Director, Communications and Marketing will exercise full functional control and 
provide support services in communications planning and delivery. 


In addition, the recommended governance of the BTP includes nine core elements: the Ministry 
Management Committee, the Project Sponsor, the BTP Steering Committee, the Project Leader, 
the Project Executive Director; the Project Director; the Quality Council; the Contract and Risk 
Manager, and, the Internal Audit Mandate. The roles of each of these core elements are set out in 
Sections 3.4.1 to 3.4.9 inclusive. 
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3.4.1. Ministry Management Committee 


The Deputy Minister of MCSS has administrative and operational accountability for the 
Ministry. Because BTP is integral to the core business of MCSS, the Deputy Minister exercises 
accountability for BTP’s outcomes through the Management Committee where significant direct 
and collateral impacts and risks can be determined and resolved. 


Members of the Management Committee with direct or collateral interests and obligations 
regarding BTP will specify those interests and obligations in Memoranda of Agreement with the 
BTP Project Office. The Memoranda of Agreement will include, for example, the relevant 
objectives, roles, responsibilities (e.g., the exercise of functional control) and undertakings (e.g., 
the provision of services or resources to the BTP) of each division head at the Management 
Committee table in committing to the success of the BTP. The Deputy Ministers’ concurrence to 
each Memorandum of Agreement is necessary. 


The Management Committee has, as a standing agenda item for review monthly, the status of the 
BDP Project Plan including schedule analysis, budget analysis and project resourcing. The 
MCSS BDP Executive Director is responsible to present this monthly status report. 


3.4.2. Project Sponsor 


The Project Sponsor is responsible for ensuring that the Ministry understands the value and 
importance of BTP. Ultimately, the Project Sponsor is accountable for realizing the benefits 
forecasted for it. Because BTP is integral to the core business of MCSS and has such pervasive 
multi-divisional and intergovernmental impacts, the Deputy Minister is the Project Sponsor. 


3.4.3. BTP Executive Committee (formerly the “Steering Committee” in the 
Agreement) 


The mandate of the BTP Executive Committee is to provide regular executive guidance, in the 


form of both approval and advice, to the BTP Project team. The Committee’s responsibilities 
include: 


e Ongoing review and acceptance of the BTP master plan; 


e Annual acceptance of the Task Orders to be undertaken in each fiscal year and review 
when: 


- scope is significantly changed, increased or decreased and/or 
- projected cost is increased by more than 5% and/or 


- schedule for delivery is delayed by more than two weeks 


e Commitment of resources from the Parties to fulfill Task Orders; 
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e Arbitrate disputes between the Project Directors, including the option to use senior 
management and an independent third party in accordance with Section 19 of the 
Agreement; 


e Resolution of issues raised by the Project Management Office for the Steering 
Committee’s attention; 


e Identification of issues requiring the establishment of strategic direction across 
MCSS; 


e Participation in the Project’s risk and quality management processes; 
e Implementation of Quality Council recommendations; 
e Senior management linkages to broader MCSS and government planning processes; 
e Review and monitoring of the Financial Arrangement. 
The composition of the Executive Committee (formerly the “Steering Committee”) is: 


Chair: Assistant Deputy Minister (ADM) Social Assistance & 
Employment Opportunities (SA&EO) 


MCSS Members: ADM Program Management 
ADM Business Planning and Corporate Services 


Andersen Members: Managing Partner, Government Services Canada 
Managing Partner, Ontario Government Services 


Project Directors: BTP Project Executive Director 
BTP Project Director 
Andersen Project Director 
The Chair of the Executive Committee is accountable to the Deputy Minister of MCSS. 
The Executive Committee will meet at the discretion of the Chair. A summary status report will 


be prepared by the Project Management Office and distributed to all Committee members bi- 
weekly. The Committee will define its own approval protocols. 


3.4.4. Project Leader and Chair, Executive Committee 


On behalf of the Deputy Minister, the Project Leader (the Assistant Deputy Minister, Social 
Assistance and Employment Opportunities) has executive responsibility for BTP, providing 
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direction within delegated authorities related to: the performance of the approved project plan 
and budget; and, interdivisional and intergovernmental coordination. Such coordination is 
basically facilitated within MCSS through Memoranda of Agreement. 


3.4.5. Project Executive Director 


Reporting to the Project Leader, the MCSS Project Executive Director is accountable specifically 
for achieving the defined project objectives within the approved plan and budget. In this regard, 
the Project Executive Director is responsible for the effective management of the relationship 
with Andersen Consulting (AC). Effective relationship management embraces those concepts 
and specifications set out on pages 48, 49 and 50 of the Management Board’s Secretariat’s 
Guidelines for Common Purpose Procurement dated October 1998. Effective relationship 
management also includes the optimum balancing of MCSS and AC resources to maintain the 
approved project plan and budget and the exercise of quality control on the resources provided by 
AC and MCSS. The Project Executive Director must ensure that the project team has all the 
necessary competencies required by BTP. This responsibility requires the careful determination 
of the factors that demand particular skills or expertise, whether sourced from MCSS or AC, then 
staffing the project to include them. On behalf of the Project Leader, the Project Executive 
Director manages all external aspects of the BTP to facilitate the achievement of BTP objectives. 
The Project Executive Director is the primary contact for the AC Project Director and the AC 
Partner-in-Charge. The Executive Director’s specific duties include: 


e Provide strategic leadership to the Project including management of the AC 
relationship; . 


e Obtain appropriately skilled staff to fulfill the requirements of each Task Order; 


e Communicate the Project goals, operational and organizational philosophies, policies 
and progress — within the Project, within MCSS and to external stakeholders; 


° Manage the Financial Arrangement defined in Schedule “B” of the Agreement; 

e Represent the Project to all organizations, and in all activities, carrying the full 
authority and responsibility of the Project as it pertains to the obligations and 
responsibilities of the Project; 


e Approval of all Task Orders and change orders; 


e Approval of Service Level Agreement Key Milestones with respect to Critical Task 
Orders; 


e Review and approve updates to Project management (Schedule “D”) and to the 
Project management Task Order (Schedule “E”) as appropriate, including submission 
of Project management change orders to the Steering Committee as appropriate; 


ee ce ee 
Hickling Lewis Brod Inc Page 28 


e Acceptance of Project Deliverables in accordance with Section 9 of the Agreement; 
e Participation in MCSS management structures (e.g., the Steering Committee); 


e Sharing information regarding Common Purpose Procurement (CPP) to other 
government agencies; 


e Bi-weekly project status reporting to the Ministry Management Committee. 


As the BTP moves through the current design phase and into the so-called “Build” and “Roll- 
Out” phases, the balance of project resources should begin shifting toward an increasing 
proportion of MCSS staff particularly to manage and execute the roll-out of the new systems and 
processes and to assume their operational control within the context of the permanent MCSS 
structures. It is this growing dependence on MCSS resources and concomitant operational and 
change management demands that lead one to conclude that a Project Director is a necessary 
addition to the MCSS team. Thus, a “two in a box” leadership structure — the Executive Director 
and the Project Director — is proposed. 


3.4.6. Project Director 


Reporting to the Project Executive Director, the Project Director, whose primary skills and 
experience relate to change management in the MCSS environment, is responsible for the 
management of all internal aspects of the BTP. This person ensures that the appropriate qualified 
resources are effectively deployed and managed.to achieve the BTP objectives. The Project 
Director stands in for the Project Executive Director and, reciprocally, represents critical 
continuity in the absence or incapacity or either party. The direction and control of all MCSS 
resources deployed in the BTP are the responsibility of the Project Director as is the day-to-day 
working relationship with AC project personnel. The Project Director’s specific duties include: 


e Manage the day-to-day project activities to accomplish the Project’s mission within 
the scope of the Project and the Project master plan; 


e Execute the Project management processes; 

e Provide direction and guidance necessary to ensure successful implementation of the 
Project in all functional areas, to facilitate the establishment of proper interfaces, 
completion of critical assignments within accepted quality objectives, on schedule 
within the allocated budgets; 

¢ Designate specific authorities for the assign resources to Task Order planning; 


e Ensure Service Level Agreements for Critical Task Orders are developed as required; 


e Recommend approval of all Task Orders and change orders; 
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© Human resource performance management of all MCSS and Municipal personnel 
assigned to the Project; 


e Recommend acceptance of Project Deliverables in accordance with Section 9 of the 
Agreement; 


e Liaison with SA & EO directors and other areas of MCSS (including the Information 
Systems Branch on issues dealing with CIMS and MAIN); 


e Government financial tracking; 

e Reporting requirements to Management Board Secretariat; 
e Responding to FIPPA requests as appropriate. 

3.4.7. Quality Council 


The Quality Council provides advice to the Project Sponsor regarding the needs of the BTP 
stakeholders, inside and outside of MCSS, and the appropriateness of the means available to 
satisfy those needs. The Deputy Minister, as Project Sponsor, chairs the quarterly meetings of 
the Quality Council. 


The Quality Council will be comprised of MCSS appointed and Andersen quality leaders, and 
include municipal membership. The Quality Council will operate in a complementary way to the 
Project’s Quality Management function. Whereas Quality Management focuses on day-to-day 
delivery of stakeholder expectations, the Quality Council provides independent oversight and 
identifies future potential quality issues. Through periodic quality reviews the role of the Quality 
Council is one that: 


e Ensures that the Project stays focused on the needs and expectations of the Parties by 
evaluating the execution of the Project’s quality plan; 


e Validates that current, and future, commitments made by the project are achievable; 


e Provides the Project Directors with coaching and independent advice at arm’s length 
from the Project. 


3.4.8. Contract and Risk Manager 


Reporting functionally to the ADM, Business Planning and Corporate Services and operationally 
to the BTP Project Executive Director, the Contract and Risk Manager will ensure that the terms 
of the agreement dated January, 1997 between Andersen Consulting and MCSS are met and that 
the Crown’s interests are protected. In addition, the Contract and Risk Manager will maintain a 
current list of identified risks arising from the BTP Business Case and Risk Analysis. From that 
list, the Manager will continuously analyze the risk mitigation actions of the offices of primary 
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and collateral interest in specific risk areas and report progress to the Project Executive Director. 
In his contract management role, the Manager will recommend payment under the agreement 
when: the work has been performed; the goods supplied or the service rendered; and, the amount 
is consistent with the terms of the agreement. On behalf of the Project Executive Director, the 
Manager will be the first level of contract dispute resolution before engaging the formal dispute 
resolution mechanisms contemplated in the agreement. 


3.4.9. Internal Audit Mandate 
In co-operation with the Contract and Risk Manager and with the approval of the Assistant 
Deputy Minister, Corporate Services and Business Planning, the MCSS Internal Auditor will, 


based on the current risk management profile of the BTP, include periodic audits of the risk 
mitigation measures and outcomes. 


3.5. Relationship Management 


Review Recommendation 14 — Relationship Management * 













Within the governance structure recommended for BTP, a key to the effectiveness of the 
relationship between MCSS and Andersen Consulting is the quality of communications between 
the two parties and the decision-making that arises therefrom. Just as the project plan sets 
rigorous performance standards for performance of the work itself, so should the parties set 
performance standards for communications (status reporting, issue resolution, etc.) and decision- 
making (resource deployment, work product approvals, task order approvals, acceptance, etc.) 
more aggressive (timing, frequency, escalation, etc.) than those found in the current Agreement. 


The better are day-to-day communications, typically, the better is the problem solving and 
decision—making. In large complex projects, frequent communications events are a tried and true 
model for success as is the maintenance of records of decisions and a list of unresolved issues. 
Agendas for meetings at any level are helpful so long as an element of open-endedness is 
allowed. 


In addition, insofar as delays in decision-making can impede the progress of the project schedule 
and deliverables, the escalation process should not tolerate delays at each level of the decision- 
making or escalation chain assuming, of course, that adequate staff work (the responsibility of 
the Project Management Office) has been completed for the decision-makers. Notably, a key area 
of communications is the management of changes to the project whether in project management, 
metrics, design, etc. The change management process set out in the project management regime 
for the project is a good one and should be performed by the parties rigorously as a key success 
factor. 
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3.6. Automating Social Assistance Project (ASAP) and the BTP 


The degree and specific nature of the interdependence between ASAP and BTP will be knowable 
when the current BTP Service Delivery Model (SDM) design requirements are known in the late 
summer of 1999. It is anticipated that most of that interdependence or value transfer between the 
two projects will be infrastructural not functional. In other words, in terms of outcomes, they are 
two separate and distinct projects. Their respective significance as management challenges 
watrants focussed attention. At the moment, the BTP Project Director has responsibility for both 
projects. The explanation for this condition appears to be historical. That is, the previous Project 
Director brought ASAP into BTP because, at the time, it was part of the Business and 
Technology Integration Branch. 









Review Recommendation 15 -- Automating Social Assistance Project (ASAP) = 





As the project enters its third year, the scope and significance of BTP is much better known and 
the need for focussed management to achieve its outcomes is palpable. Similarly, ASAP’s needs 
are significant and worthy of focussed management attention in their own right. However, 
ASAP is dealing with the “now” needs of MCSS while BTP is the “future” of MCSS program 
delivery. Accordingly, it is recommended that: 


e ASAP should report to the Assistant Deputy Minister, Business Planning and Corporate 
Services; and 


e BTP should continue to report to the Assistant Deputy Minister Social Assistance and 
Employment Opportunities as is now the case. 


These two executives should, by formal agreement, ensure that current and future 
interdependencies are identified and attended to ina timely and responsive manner. 
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4. THE BUSINESS CASE 
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The business case analysis of BTP indicates both (i) a strong probability of positive returns to the 
taxpayers of Ontario, and (ii) a material risk of negative returns unless key uncertainties are 
managed and their effects mitigated. 


4.1. Expected Business Returns 


Expected business returns -- returns arising under central forecasts of BTP costs, schedule and 
performance -- may be summarized in terms of (i)life-cycle net benefits; (ii) return on 
investment; (iii) the economic value of social benefits; (iv) net benefits at “steady-state” (full 
BTP functionality); and (v) the effect of cost sharing with the private sector. 


4.1.1. Net Benefits 


As shown in Table 1, the realization of central cost, schedule and performance projections means 
that the BTP investment program would yield $297.2.0 million in benefits net of all costs (in 
present-day dollars of 1996 purchasing power over the 1996-2003 life-cycle of the CPP contract, 
discounted at five percent per year). Benefits arise in the form of (i) productivity gains in the 
MCSS administration of OW and ODSP, and (ii) reductions in the volume of inaccurate 
payments under these programs. 


TABLE 1: BTP LIFE-CYCLE BUSINESS CASE RESULTS (Benefits minus Costs over 
the Period April 1996 to March 2003, Excluding Social Benefits) 


( Present-day value, at five percent discount, in millions of current dollars) 





Notes: 

1) Base caseload of 591,000 cases in 1996/1997, using September 1998 forecast. 

2) Labour rates include annual escalation factors of 8% for Andersen Consulting and sub-contractors, and 5% for 
MCSS and sub-contractors (applies to costs and benefits). 

3) Cost information based on Andersen Consulting supplied workdays. 

4) Schedules developed during January 28 Session. 

5) 30% of the active caseload is assumed to have an overpayment under base and BTP figures. 


As shown in Table 2, the business benefits of BTP are manifest in improved quality of service to 
OW and ODSP recipients. 
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TABLE 2: BTP SERVICE QUALITY EFFECTS (at Steady-State) 


bility Decisions Made Correctl 
Percent of Payments Made in the Correct 
Amount 
Percent of Social Assistance Payments Made on 
Time 





4.1.2. Return on Investment 


Central projections indicate a cumulative seven-year return on BTP investment of 222 percent 
(Table 3). High returns of this magnitude indicate that modernizing MCSS business processes is 
sharply overdue. Had such investment been made earlier, returns would have been lower, yet 
nevertheless acceptable, while Ontario taxpayers would have not have waited a protracted period 
of time for greater effectiveness and efficiency in the administration of social assistance. 


TABLE 3: BTP RETURN ON INVESTMENT (excluding social benefits) 
(in percent) 
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4.1.3. Economic Value of Social Benefits 


The results outlined above account for gains in productivity and reduced program outlays. 
Central forecasts indicate that BTP will also yield an additional $49.5 million in net social 
benefits to recipients of OW and ODSP over the first seven years of the program (1996 to 2003, 
in present-day value at five percent discount). These benefits arise in the form of less delay for 
clients in applying for and obtaining legitimate social assistance. Client time savings have 
economic value because they facilitate less delay in obtaining assistance and more ready access 
to employment opportunities. When the economic value of these social benefits is combined 
with gains in MCSS productivity and spending reductions, total net benefits rise to $346.7 
million ($297.2 million plus $49.5 million). 
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4.1.4. Steady-State Benefits 


The results above are reported on a life-cycle basis so as to account for lower benefits during the 
multi-year roll-out and ramp-up of BTP systems and business processes. Central projections 
indicate that BTP processes will be operating in “steady-state” (full functionality and 
productivity) by year eight of the program (2003-4). Table 4 indicates that at that time annual 
benefits (excluding social benefits) will have reached $718.8 million while the major fraction of 
BTP costs will have been incurred in previous years. Steady-state net benefits will thus reach 
$694.3 million by year eight of the program. 


It is noteworthy that the roll-out of “early opportunity" investments (CR-1, CR-2 and CVP) has 
generated positive net benefits on an annual basis (for 1997/98 and 1998/99). While going 
forward, the BTP investment program can expect two years of negative returns, the early 
accumulation of positive outcomes means that cumulative rates of return remain positive in each 
year of the seven-year development and roll-out program (see Table 3). 


TABLE 4: BTP ANNUAL AND STEADY-STATE (EIGHTH YEAR) COSTS, BENEFITS 
AND NET BENEFITS (excluding social benefits) 
(in millions of current dollars) 








718.8 





439.8 
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4.1.5. Cost Sharing with the Private Sector. 





The results outlined above account for all BTP costs and all resulting BTP benefits. However, 
present CPP financial arrangements between MCSS and Andersen Consulting entitle the 
consultant to benefits payments of no more than $180 million, an amount that may lie beneath 
the consultant’s cost of delivering total BTP functionality at its present billing rates . If 
Andersen Consulting nonetheless delivers total functionality within the $180 million payment 
“Cap,” net benefits from the perspective of the provincial taxpayer will exceed the $297.2 
million value cited above by the difference between the Cap and Andersen Consulting’s cost. 
From the perspective of total costs, however, regardless of whether they are borne by the 
taxpayer or Andersen Consulting, the net present value remains unchanged. 


4.2. Risk Analysis 


A number of material risks confront BTP management in seeking to achieve the central forecasts 
reflected above and thus deliver positive business returns. These include (i) schedule risk; (11) 
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the risk of failure to achieve targeted levels of effectiveness; (iii) the risk of inaccurate case load 
forecasts; and (iv) the risk of inaccurate assessments of the opportunity cost of capital. 


Schedule Risk. A dominant risk is that of slippage in the roll-out and ramp-up of key processes 
and support technology. Analysis of potential slippage risk, taken together with the effects of all 
other risk factors, indicates a five percent probability that that BTP costs will exceed BTP 
benefits by more than $28 million, and a one percent risk that costs will exceed benefits by more 
than $117 million — (Table 1 -- in present-day dollars of 1996 purchasing power over the 1996- 
2003 life-cycle of the CPP contract, discounted at five percent per year). On the other hand, 
sound and aggressive risk management of the schedule is found to offer substantial rewards. As 
compared with the baseline schedule, each month of accelerated roll-out and ramp-up yields five 
percent greater net benefits. While aggressive risk management and mitigation of schedule risk 
itself carries a cost, the business case for such activity is very high. 


Effectiveness Risk and the Crystallization Process. Benefits in the form of improved 
productivity and reduced program outlays arise when BTP business processes trigger more 
effective MCSS performance in administering intake and eligibility monitoring and benefits- 
management activities. Delay in achieving provably greater effectiveness will delay, and 
weaken, business case returns. 


Multivariate statistical analysis indicates that improved effectiveness can be confirmed in relation 
to the introduction CR-1 and CR-2 in September 1997 and that provable business and social 
benefits are accruing accordingly. On the other hand, statistical analysis does not yet confirm 
that improved effectiveness has arisen from CVP, as introduced in April 1998. Since a full year 
or so of experience may reasonably be required in order to ascertain such proof, the absence of 
confirmed CVP benefits as of November 1998 is notable though not alarming. 


On the other hand, failure to ascertain the emergence of such proof with eight months of CVP 
experience should be viewed as an emerging risk and managed accordingly. Since at present the 
process of crystallizing effects and benefits, while rational and well documented, does not 
include tests of statistical proof. Such tests should be incorporated into the process as part of an 
improved risk management framework. 


Client Base. The client base is a risk factor because the benefits of BTP are found to fall more 
quickly than BTP costs as the number of active cases declines (or rise more quickly with growth 
in the number of active cases). Although case load would need to decline substantially more 
than projected under even the most optimistic economic outlook in order to materially dilute the 
business case for BTP as it is currently scoped, risk managers should recognize the link between 
the net benefits of BTP and the size of the case load and seek opportunities to manage the timing 
and scope of project outlays accordingly. 


The Opportunity Cost of Capital. Results reported above assume a provincial opportunity cost 
of capital of five percent (in dollars of current purchasing power). While this is consistent with 
current market conditions, governments at times apply above-market discount rates in appraising 
public investments as a means of guarding against the risk of crowding-out more productive 
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private sector investments. At a ten percent discount rate for example, the expected net present 
value of BTP reported in Table 1 falls from $297.2 million to $216.7 million. 


Economic reasoning supports the lower discount rate in the case at-hand, however, for two 
reasons. First, the willing participation of a private sector firm in sharing the investment risk of 
BTP indicates that crowding out is not occurring. Second, economic theory supports a lower 
discount rate for social investments on the grounds that capital markets understate the rate at 
which society is willing to defer consumption in order to sustain a minimum state of health and 
nutrition for those in need. 


4.3. Opinion 


HLB Inc. believes that the BTP investment program, in its present state of design and scope, 
offers a sufficiently high probability of positive business case returns and sufficiently low risk of 
negative returns to justify the magnitude of investment funds budgeted for the acquisition of 
associated goods and services. 


HLB Inc. analysis indicates the need for aggressive steps to manage risk and mitigate against the 
effects of uncertainty, particularly in relation to, (i) adherence to the scheduled roll-out and ramp- 
up of BTP business processes and supporting technologies, and (ii) timely statistical proof of 
system effectiveness and crystallized benefit. Failure to manage and mitigate these risk poses a 
material risk that BTP will yield negative business case returns. 
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